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ABSTRACT 

Some results from an international study of reading 
are discussed in this paper. Subjects were 10-year olds, 14-year 
olds, and secondary school seniors in 15 countries. A reading 
comprehension test was administered and the 10 and 14-year olds were 
given a speed reading test. Data analysis was at three levels: 
individual pupil scores, school characteristics associated with 
reading achievements of its students, and between— country 
comparisons. Data for all countries are presented, and compared v,ith 
means and standard deviations of American subjects. (Charts and 
tables present detailed results of the study.) (WR) 
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For ray part in today* s proceedings. I pi an to talk mostly about some aspects of 
the results from our IEA studies of reading. In the course of talking about these 
results, I shall probably have occasion to consider with you some of the methodo- 
logical problems that arise in the sort of study in which we have been engaged, but 
I will discuss these in the context of the materials and the findings on reading 
comprehension and reading speed. 



Dr, Postlethwaite has taken responsibility for telling you something about the 
development of the instruments for the latest of the ISA. surveys. In the case of 
reading, the instruments of primary concern were two. One of these was a reading 
comprehension test of the conventional type, in which the examinees read a passage 
ar.d answered multiple choice questions upon it. Each level of the comprehension 
test was made up of two sections composed of four passages and 20 to 3® test items 
depending upon the level of pupil Involved. As you have already heard, we have 
worked with 10-year-olds l4 ~y e ar - old s and individuals in the last year of secon- 
dary school. We speak of these as Populations 1, 2 and 4, and in what follows 
some of the materials and discussion will be couched around these three populations : 
Population 1 - the 10-year -olds , Population 2 - the 14 -year-olds , and Population 
4 - the group at the end of secondary education. 



In addition to the reading comprehension test, the examinees in Populations 1 and 2 
were also given a short reading speed test. This consisted of very short para- 
graphs each one made up of three sentences, the third of which was a question. 

The question was to be answered by ’underlining one of three words. The successive 
paragraphs dealt with a continuous story about a boy and a dog who walked out 
into the field, what they saw and what they did. The test was intended to be 
very easy reading, and the questions were designed to be answered correctly by 
9C$ or better of students. The little test consisted of 40 of these paragraphs 
arid was given with a four -minute time limit. 







We had hoped to include some study-type reading in which an extended passage is 
studied and removed before testing, and to get evidence on this type of performance , 
but practical limitations of time, both to produce materials and to apply them to 
the students, led us eventually to abandon this and restrict ourselves to the com- 
prehension test and the speed test. 

As you have also heard, we had as a supplementary measure a Same -Opposite vocabu- 
lary test of 4 q words at each level. This served as a simple descriptor of word 
knowledge and vocabulary size, but will play relatively little part in what I world 
like to talk about today. 

Dr. Postlethwaite has talked in general about the test development, but perhaps 
I should amplify a little bit some of the special problems in connection with 
reading tests. The most obvious one is the problem of language. Among the 15 
countries that participated in the administration of the reading tests, 12 dif- 
ferent languages were represented. Operationally, this meant that when test 
materials were initially supplied in some other language they had to be translated 
into English as a common medium of communication among the countries. They were 
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reviewed and criticised by each national committee with regard to their appropri- 
ateness to that country and with regard to the technical aspects of the passages 
and such items as were initially available on them. Items were written in English 
and were reviewed by individuals from the different countries. Each passage was 
translated into at least two languages other than English for preliminary try-out, 
as part of the development of the test materials. The final passages and items 
were selected on a basis of the item statistics from generally three and sometimes 
four different countries, and final versions of the passages and items were pre- 
pared— but still in English, At this point , all materials had to be translated 
into the language of each of the participating countries. We were encouraged that 
( this could be done without seriously disrupting the character of the reading 
'materials on the basis of our previous experience in the earliest study that we 
did back in 19 & 2 , in which it appeared that not only general level of difficulty 
but even the difficulty of specific items showed good stability from language to 
language. 

Each national center was responsible for handling the translation into its own 
language, and each national center was encouraged to provide an independent trans- 
lation back into English by a separate translator to see to what extent the charac- 
ter of the original material had been maintained through the process of translation. 
However, both time and resources have been limited, and we received at most a back- 
translation of a sampling of the passages and items. Most of the participating 
countries felt that their limited resources were better spent on doing dual trans- 
lations from English into their ,wn language and trying to maximize the precision 
and quality of the forward tra n sl a tion, and consequently any evidence from a re- 
translation as to the precision with which the original passage had been maintained 
in translation is quite limited. 

We also have data for the present tests concerning the consistency of item dif- 
ficulty and other item statistics from country to country, and we have data on the 
popularity of the error choices in different countries for each item, but this 
material is voluminous and has not yet been very thoroughly examined, I can say 
in general that the correlations of item difficulties across languages are quite 
substantial, with the possible exception of those countries in which students 
had a very great deal of difficulty with the material and were responding to sub- 
stantial numbers of the items at a chance or near chance level. 

Cross -national studies have two unique contributions to make to our understanding 
of educational phenomena. On the one hand, one is interested in parallel analyses 
within each country, in which case the countries serve as replications of an experi- 
ment and provide an opportunity to test in a broader context relationships that 
have been previously observed in a single national system. It is also possible to 
make comparisons across countries, seeing in what respects countries differ in 
their performance and attempting to understand these differences in terms of the 
characteristics of the countries involved. We have repeatedly stated that we are 
not interested in an intellectual Olympic games, but in using national differences 
in educational practices and procedures as a quasi -experiment in different educa- 
tional treatments. Hence, the kinds of analysis which I shall present today will 
be at three levels. On the one hand, there will be analyses at the level of the 
individual pupil. We have a number of types of data which characterize the pupil 
as an individual, and it is possible to study within each country individual cor- 
relates of reading achievement. At the second level, we will be interested in 
analyses in which the school is the unit; we will be trying to identify those 
characteristics of the school as an institution that are associated with the read- 
ing achievements of its students. Here, it soon becomes apparent that the most 
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Important determiners of the reading achievement of the pupils in a school are 
the characteristics of the pupils themselves in terms of their backgrounds and 
out -of -school resources* One of the sticky problems to which we will need to 
give some attention is the problem of identifying school effects, as distinct 
from the input characteristics of the students that represent the clientele of a 
particular school* Finally, as X said earlier, we are interested in analyses that 
use the country as a unit and compare characteristics of performance with other 
variables from country to country * In what follows , I want first to t alk about 
between-country differences, then to talk about between-pupil differences and 
fi nall y to talk about between-school differences* 

As I talk about the test performance of the pupils in different countries, I shall 
identify only the results for the United States, When the full results appear, 
some analyses by country will be made in which individual countries will be iden- 
tified, but it is not our purpose either to applaud or point the finger at any 
country in relation to its achievements. We are interested only in understanding, 
and in this context it is the nature of the country and its econ omi c, social and 
educational system that we are interested in trying to relate to the achievements 
of school pupils. 



As a very first question, we might ask how big the differences in reading test 
performance were from country to country at the three educational levels at which 
we were working. In order to provide a kind of frame of reference for looking at 
the size of these differences and at their direction in relation to performance 
in the United States, X have scaled each country in relation to the mean and 
standard deviation of the United States. That is, the United States defines the 
baseline, and deviations above or below that baseline are expressed in standard 
deviations of the USA distribution* (Parenthetically, I might say that the varia- 
bility of performance in the United States is either uniquely the largest or tied 
for largest of any of the countries involved. The heterogeneity of reading per- 
formance is very great here.) 

I call your attention to Chart 1 in the handout that has been prepared, j/hich shows 
the amount and direction of the differences of the various countries from the USA 
Daseline or USA par. You will notice that in Populations 1 and 2, th* means for 
a large number of the countries cluster rather closely around the United States 
mean and the United States seems to be fairly representative of this large cluster 
of primarily European or European-oriented countries. These are typically econom- 
ically and industrially developed countries with a practice of universal education 
for individuals up through the age of 14. 

Included in our study this time were three relatively underdeveloped countries, 
and their lack of economic development and of a background of universal education 
shows up very dramatically in their performance on the tests. These are the three 
countries that fall very clearly at the bottom of the distribution of average 
reading scores in all three of the populations that we studied. You will notice 
that the discrepancy is a full standard deviation or more in terms of the United 
States distribution and that these countries deviate much more dramatically from 
the baseline than do any of the other countries. 

It is clear from this chart that the United States shows up very much less well 
at the end of secondary school than at the two earlier levels. This is a familiar 
phenomenon and we all recognize that it results in substantial measure from the 
much higher preportion retained in school through the end of secondary education 
in the United States than in the other countries with which we had to deal. This 
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Naturally, the first place one looks for information that will predict the per- 
formance of the individual is in the social and cultural background of the child 
in his home and family. We had several indicators of home and family status. The 
information on these was undoubtedly much less than perfectly accurate, since it 
was furnished by the children themselves in response to a questionnaire dealing 
with a multitude of facts about their background and schooling, and at least in 
some countries furnished by children who would certainly have had very great dif- 
ln T readin S the questionnaire, in view of their obviously limited reading 
ability. In Table 3, I have listed several of the home background predictors 
hat were most effective, and have shown the range of correlations for these across 
United Stat ^ k e ^ ore > time I identify the relationship only within the 
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resources such &£ libraries and librarians , books in "the classroom and 
so forth. However, we were not able to get any detailed picture of the classroom 
materials or instructional procedures excepting as these were reported in general 
terms by the teachers. 

A second difficulty arises from the very large number of specific bits of informa- 
tion that are generated by a questionnaire that seeks through many indicators to 
characterize a school program. We found that the several questionnaires generated 
literally hundreds of items of information about each school, each one being a 
weak little indicator of something about the school situation* We ended up with 
substantially more predictors than we had schools in any given country, and we 
faced a very serious problem of degrees of freedom in an analysis of this sort. 

We needed in some way to reduce the number of variables to a manageable size. 

We encountered a third and perhaps most serious difficulty in that the clearly 
potent predictors of reading achievement for a given school related to the average 
quality of the pupil input in terms of the kinds of indicators that were effective 
for predicting achievement of single individuals. Thus, in Table 5 are shown the 
correlation of scaled father* s occupational level, average father* s education, 
average mother's education, and average number of books in the home with average 
reading achievement for the various countries at level 2. The median size of 
these corr slat ions is shown in the last row of the table and it can easily be 
seen from these correlations that a very substantial proportion of the variance in 
reading achievement for pupils in any given school is accounted for by factors 
that lie outside of the school *s control and that represent the life background 
from which the individual pupil has come. Under the above circumstances, it is 
only too possible that any school variables that turn up with appreciable correla- 
tions have these correlations because of the fact that they axe also related to 
the type of pupils attending that school. For the full and formal analysis of 
school variables as predictors of achievement, then, it is necessary to look not 
only at the initial relationship between the variable and the achievement measures 
but also at the partial correlation when a complex of these individual background 
TOsi^bles is part ial , l e d out. As a way of part jailing out differences in input, 
we deVeloped what one of my colleagues has called a "school handicap score" to 
represent the expected achievement based upon a composite of these individual 
background factors, and have looked at the residual correlations with achievement 
after these background factors had been statistically paxtialled out. 

There are many too many variables characterizing schools to make it either possible 
or meaningful to present them fully at this time. I have picked out a few that 
some of you might have anticipated a priori to be effective predictors of school 
achievement and have prepared in Table 6 a representation of their original 
zero-order correlations with reading achievement. There is too much detail in 
the table for you to apprehend at the present time, but it will pay sobering 
study* Let me just comment on one or two of the characteristics that you may 
find there. 

In general, educators argue for the need for a higher level of funding of education, 
and the implication is that more expenditure per pupil should yield higher levels 
of pupil achievement. You will find in the table a variable on per pupil expendi- 
ture, based upon the reported expenditure level supplied by the administrator of 
the school. The accuracy of these reports is somewhat suspect. However, as you 
examine the column of correlations for this variable, X think that you will agree 
that the relationship is puzzling and somewhat disillusioning. 
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A second point that is frequently argued is tnat smaller classes are highlv 
desirable for educational achievement. The implication is that children in 
s mall er classes should achieve more than children in larger classes. If you 
look at the column labeled mother tongue class size, you will find t ha t the 
direction of the zero-order correlations is as often the reverse of wnat this 
doctrine would propose as it is consistent with it. At the superficial level, 
the children in the larger classes often make out rather better than the children, 
in the smaller ones. You might quite reasonably argue that the relationship 
tliat we are concerned with here is not a linear one and that the linear correla- 
tion that we display is inadequate to represent it. You could also argue that 
the children who are assigned to small classes are assigned to small alasses 
because they are poor performers and that the direction of causation is not from 
class size to achievement but from achievement to class size. You could also 
point out that small classes are likely to occur in small rural schools where 
other handicaps exist that are not reflected adequately in our school handicap 
score. This is also a reasonable proposal. However, be that as it may, the 
evidence that we have gives little encouragement to expecting substantial improve- 
ments in achievement by reducing the size of the classes. 

Other elements that would typically be thought of as supporting reading achievement 
are a staff of reading specialists, a school library, books in the classroom and 
other supporting auxiliary school personnel. We show evidence on & number of 
these factors in the table. Again, there is no consistent evidence of a positive 
effect from these supporting services, and in some instances the relationships are 
fairly consistently negative. Once again, we see only relationship and not causa- 
tion, Once again, we can argue with some cogency that remedial teachers and 
other kinds of special services are concentrated in those schools where problems 
are known to exist. To this extent, a negative relationship may represent an 
adaptation of a school system to the problems that it faces. However, we find a 
great dearth of evidence that institutional arrangements designed to help young- 
sters in their efforts to read well and effectively do in fact do so. 

Generally speaking, our whole effort to identify school causes of achievement, at 
least in the reading area, has seemed a rather unproductive one— at least to me. 

At this point, I am not sure that I know fully why we have done no better than we 
have. Of course, our findings are not without precedent. The Coleman Report did 
not tend to attribute great potency to school influences upon achievement in the 
United States, and perhaps we should not have expected to find them in the coun- 
tries that we studied either. Certainly, the indicators that we have are limited 
and rather superficial ones and no one of them could be expected to carry a great 
deal of predictive weight. That is why we had hoped to merge them into compounds 
and composites that would be more effective as predictors. However, there seems 
to me to be little to compound if one looks at these variables individually. The 
results from country to country are modest and inconsistent, and one questions 
whether putting the little bits together in either an a priori fashion or an em- 
pirically determined one could add up to a very great deal . 

These last findings have been somewhat discouraging to me. I'm not sure quite what 
I'm discouraged about more profoundly — the kind of international survey in which we 
have engaged or the educational enterprise itself. Of course, we are not making a 
comparison between schooling and no schooling. We are trying to differentiate the 
effects of different types and qualities of schools within a given cultural setting. 
The variability among schools may be small relative to the impact of schooling taken 
in toto. From this point of view, variability may not be terribly important or 
terribly productive to study. However, we d<> seem to have drawn a rather* poor hand 
as far as being able to provide strong cues as to what it is about a school environ- 
ment that results in better reading achievement in the pupils thereof. 
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Chart 2 



Percent of Near-Chance Scores on Reading Comprehension 



1. Peter has a little dog. The dog ts black with a white 
spot on hi# back and one white 'teg. The color of 
Peter’s dog is mostly 

black brown gray 



2. When Peter got the dog it was a small puppy. Now 
the dog is a little more than two years old- How many 
years has Peter had the dog? 

one two three 



3. Peter’s dog has a spot on his back. That is why Peter 
named the dog Spot. The dog was named after the 
spot on his 

back ear leg 



4. The dog has learned to do two tricks. One trick is 
to oatch a ball. To stand on its hind legs is the second 

story trick way 



5. When he was a puppy Spot was fed three times a 
day. Now he is fed only once. The number of times 
is now 

often less many 



6. Spot is most happy when he gets a bone. He would 
like to have a bone every day, but he does not get 
one that 

small hungry . often 



7. Spot lives in his own little house. It is a red house 
and it is made of wood. The house that Spot lives 
in is 

green red whke 



8. Peter’s mother does not like to have Spot in her house 
when he has dirty feet. When his feet are dirty Spot 
must stay 

inside hungry outside 



9. Sometimes Peter has dirty feet too. Then his mother 
makes him wipe them off on the mat. The mat is used 
to keep the house 

clean dry warm 



Chart 3 

Reproduction of First Page of Reading Speed Test 



Table 1 



Percent of Pupils Making 3 or More Errors 
on First 9 Reading Speed Items 



IQ-Year-Olds 

4.4 
7,2 
8.0 

8.1 

9.5 

9.7 



14 -Year -Olds 

0.4 

1.3 

1.5 

1.8 

2.1 

2.2 



10.7 




turn 


mw 


i~ USA ^ 


2.8 



12.8 5.6 



USA 



22.5 

26.5 

41.5 



54.9 

61.4 



5.8 

6.1 

6.4 

19.6 

25.8 

51.5 



Table 2 



Correlation of Selected national Characteristics 
with Rational Mean Reading Score 
of 14 -Year -Olds 



Variable Correlation 



Mean level of father's education 0.59 
Mean level of mother's education 0.75 
Mean number of books in home 0.84 
Percent homes with dictionary - 0.02 

Percent homes with daily paper 0.82 
Mean number of nogaslnes in home 0.69 
Mean msaber of siblings -0.70 
Percent reporting TV watching 0.95 
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Table 4 






Girls* 




SjgeriQrity ove r Boys in Reading Comprehension Test 
( In /Standard Deviation Units ) 



lO-Year-Olds 

.146 

.036 

cm* 

.128 

.116 

.094 

.050 

.048 

.046 

.030 

.014 

•.014 

•.096 

-.107 



14 -Year-Olds 


• 

End of Secondary 


.3^0 


.392 


.192 


.348 


.116 


.264 


.092 


.170 


fToSSi 


.158 


.046 


cm 


.044 


.086 


.034 


.038 


.000 


-.038 


CM 

» 

• 

1 


—.066 


-.062 


-.118 


-.066 


-.124 


-.132 


-.326 


-.232 





Median .049 



.039 



* USA identified by boot in each case. 



.086 



Table 5 

Correlations of Average Reading Comprehensio 
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Correlation of Selected School Variable* with Avenge Rending Achievement 
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Group 



10 yr 14 yr Terminal 




Mean Reading Score by Country and Level, Expressed in Relation 

to USA Mean and Standard Deviation 
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Chart 2 

Percent of Near -Chance Scores on Reading C apprehension Test 





1. I*eter ha* a little clog. The dog is black with a white 
spot on his hark and one white leg. The color of 
IVter’s dog is mostly 



black 



brown 



gray 



2. When Peter got the dog it was a small puppv. Now 
the dog is a little more than two years old. How many 
years has Peter had the dog? 



one 



two 



three 



Peter’s dog has a spot on his back. That is why Peter 
named the dog Spot. The dog was named after the 
spot on his 



back 



ear 



leg 



4. The dog has learned to do two tricks. One trick is 
to oatch a ball. To stand on its hind legs is the second 



story 



trick 



way 



W’hen he was a puppy Spot was fed three times a 
day. Now he is fed only once. The number of times 
is now 



often 



less 



many 



6. Spot is most happy when he gets a bone He would 
like to have a hone every day, but he does not get 
one that 

small hungry often 



7. Spot lives in his own little house. It is a red house 
and it is made of wood. The house that Spot lives 
in is 



green 



red 



white 



8. Peter s mother does not like to have Spot in her house 
when he has dirty feet. When his feet are dirty Spot 
must stay 

inside hungry outside 



9. Sometimes Peter has dirty feet too. Then his mother 
makes him wipe them off on the mat. The mat is used 
to keep the house 

clean dry warm 



Chart 3 

Reproduction of First Pn^e o^ Rend •i nf’’ TV»<st 



Table 1 



Percent of Pupils Making 3 or More Errors 
on First 9 Reading Speed Items 



10-Year -Olds 

4.4 
7.2 
8.0 

8.1 

9.5 
9.7 



14 -Year-Olds 

0.4 

1.3 

1.5 

1.8 

2.1 

2.2 



10.7 






12.8 



USA 






2.8 

5.6 



22.5 

26.5 

41.5 



5.8 

6.1 

6.4 



54.9 19.6 

81.4 25.8 

51.5 



Table 2 



Correlation of Selected national Characteristics 
with National Mean Reading Score 
of 14 -Year-Olds 



Variable Correlation 



Mean level of father's education 0.59 
Mear level of mother's education 0.75 
Mean number of books in home 0.84 
Percent hones with dictionary -0.02 

Percent hones with dally paper 0.82 
Mean number of magaslnes in home 0.69 
Mean number of siblings *0.70 
Percent reporting TV watching 0.95 
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Table 4 



Girls* Superiority over Boys in Reading Comprehension Test 

(In Standard Deviation Units) 



10-Year -Olds 


14-Year-Qlds 


Snd of Seco 


.146 


.340 


.392 


.136 


.3-92 


.346 




.116 


.264 


.128 


.092 


.170 


•116 


turn 


.158 


.0^4 


.046 




.050 


.044 


.086 


.048 


.034 


.038 


.046 


.000 


-.038 


.030 


-.052 


-.066 


.014 


-.062 


-.UB 


-.014 


-.066 


-.124 


-.096 


-.132 


-.326 


-.107 


-•232 
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VO 


.039 


.086 



USA Identified by booc in each case. 
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Correlation of Selected School Variables with Avenge Reading Achievement 
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